{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exponential-Form of the variational mixture of Gaussians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.random import randn, seed, randint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making use of the identity\n",
    "\n",
    "$$\n",
    "\\text{Tr}(A^TB) = \\text{vec}(A)^T \\text{vec}(B)\n",
    "$$\n",
    "\n",
    "one can write the exponential part a mixture of gaussians in the form\n",
    "\n",
    "$$\n",
    "    \\text{vec}(\\boldsymbol\\eta)^T \\text{vec}({\\bf x}_n{\\bf z}_n^T)\n",
    "$$\n",
    "\n",
    "Where\n",
    "$$\n",
    "    \\boldsymbol\\eta = \\begin{bmatrix}\n",
    "    \\boldsymbol\\eta_1^T \\\\\n",
    "    \\vdots \\\\\n",
    "    \\boldsymbol\\eta_K^T \\\\\n",
    "    \\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "    {\\bf z}_n = \\begin{bmatrix}\n",
    "    z_{n1} \\\\\n",
    "    \\vdots \\\\\n",
    "    z_{nK}\\\\\n",
    "    \\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(314)\n",
    "k, m = 5, 10\n",
    "eta_hat = randn(k, m)\n",
    "x_n = randn(m, 1)\n",
    "\n",
    "z_n = np.zeros((k, 1))\n",
    "z_n[randint(0, k - 1)] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "    {\\bf z}_n \\bar{\\boldsymbol \\eta}{\\bf x}_n\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.01147608]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z_n.T @ eta_hat @ x_n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.01147608]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eta_hat.T.reshape(-1, 1).T @ (x_n @ z_n.T).reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "    \\sum_{k=1}^K \\text{vec}(\\boldsymbol\\Lambda_k)^T \\text{vec}\\left({\\bf x}_n{\\bf x}_n^T\\right) z_{nk} = {\\bf z}_n^T \\boldsymbol\\Lambda\\text{vec}\\left({\\bf x}_n{\\bf x}_n^T\\right) = \\text{vec}(\\boldsymbol\\Lambda)^T \\text{vec}\\left(\\text{vec}\\left({\\bf x}_n{\\bf x}_n^T\\right) {\\bf z}_n^T\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(3141)\n",
    "lambda_ks = np.random.randn(k, m, m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-10.056419480123484"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((lambda_ks.reshape(k, -1) @ (x_n @ x_n.T).reshape(-1, 1)) * z_n).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-10.056419480123486"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(z_n.T @ lambda_ks.reshape(5, -1) @ (x_n @ x_n.T).reshape(-1, 1)).item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec_gram = (x_n @ x_n.T).reshape(-1, 1)\n",
    "Lambda = lambda_ks.reshape(k, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-10.056419480123484"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.trace(Lambda @ vec_gram @ z_n.T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-10.05641948]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lambda.T.reshape(-1, 1).T @ (vec_gram @ z_n.T).reshape(-1, 1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
